Analysis and feedback system for personal care routines

ABSTRACT

A computer system obtains digital video data of at least one physical consumer product (e.g., two or more cosmetics products) in a personal care routine; analyzes the digital video data (e.g., using automated object recognition or gesture recognition techniques); detects at least one physical interaction with the at least one physical consumer product (e.g., two or more applications of cosmetics products) based at least in part on the analysis of the digital video data; and causes customized personal care routine data (e.g., a computer animation simulation or a comparison of the user&#39;s routine with routines of other users) to be presented in a user interface. The customized personal care routine data is based at least in part on the at least one physical interaction.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features ofthe claimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

In one aspect, in a computer-implemented method, a computer systemobtains digital video data of at least one physical consumer product(e.g., two or more cosmetics products) in a personal care routine;analyzes the digital video data (e.g., using automated objectrecognition or gesture recognition techniques); detects at least onephysical interaction (e.g., two or more applications of cosmeticsproducts) with the at least one physical consumer product based at leastin part on the analysis of the digital video data; and causes customizedpersonal care routine data to be presented in a user interface (e.g., ofa client computer system). The customized personal care routine data isbased at least in part on the at least one physical interaction. In anembodiment, the step of causing the customized personal care routinedata to be presented in the user interface comprises generating thecustomized personal care routine data based at least in part on the atleast one physical interaction and transmitting the customized personalcare routine data to a client computer system for presentation in theuser interface.

In an embodiment, the method further comprises obtaining productinformation for the at least one physical consumer product, wherein thecustomized personal care routine data is further based on the productinformation. In such an embodiment, the product information may beextracted from a graphical code on the at least one physical consumerproduct.

In an embodiment, the method further comprises representing the at leastone physical consumer product as an object in a personal care routinesimulation engine; and representing the detected physical interaction asan interaction with the object in the personal care routine simulationengine. In such an embodiment, the customized personal care routine datamay include a computer-animation simulation of a personal care routinegenerated by the personal care routine simulation engine.

In another aspect, a client computer system (e.g., a smartphone, tabletcomputer, or other suitable computing device or set of computingdevices) comprises a camera; a processor; and computer-readable mediahaving stored thereon instructions configured to cause the clientcomputer system to perform steps comprising capturing digital video dataof at least one physical consumer product (e.g., two or more cosmeticsproducts) used in a personal care routine; obtaining customized personalcare routine data (e.g., a computer-animation simulation of a careroutine or a comparison of the user's routine with other users); andpresenting the customized personal care routine data in a user interfaceof the client computer system. At least one physical interaction withthe at least one physical consumer product (e.g., two or moreapplications of cosmetics products) is detected based at least in parton automated analysis of the digital video data (e.g., using objectrecognition techniques), and the customized personal care routine datais based at least in part on the at least one physical interaction. Inan embodiment, obtaining the customized personal care routine datacomprises transmitting the digital video data to a remote computersystem and receiving the customized personal care routine data from theremote computer system, wherein the at least one physical interaction isdetected by the remote computer system based at least in part onanalysis of the digital video data performed by the remote computersystem, and wherein the customized personal care routine data isgenerated by the remote computer system. In an embodiment, theinstructions are further configured to cause the client computer systemto capture a digital image of a graphical code on the at least onephysical consumer product, or obtain product information from the atleast one physical consumer product via short-range radio-frequencycommunication.

In an embodiment, a non-transitory computer-readable medium includesinstructions configured to, when executed by one or more computingdevices of a computer system, cause the computer system to perform anyof the functions described herein, in any combination.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic diagram that provides a high-level overview of asystem in which various embodiments of the present disclosure may beimplemented;

FIG. 2 is a block diagram that illustrates an example embodiment of aclient computing device according to various aspects of the presentdisclosure;

FIGS. 3 and 4 are flowcharts that illustrate example embodiments ofmethods according to various aspects of the present disclosure; and

FIG. 5 is a block diagram that illustrates aspects of an exemplarycomputing device appropriate for use with embodiments of the presentdisclosure.

DETAILED DESCRIPTION

Consumers often use many different consumer products in various ways ina daily personal care routine. For example, a consumer may apply severaltypes of cosmetics and personal care products at home in order toprepare for her work day. The number of ways in which consumer productsmay be combined and used in a personal care routine is potentiallyunlimited. The possible variations in products and how they are used canresult in substantial differences in the consumer's satisfaction andexperience. Even if two consumers use the same set of personal careproducts, the order, amount, and time spent applying each product canvary dramatically from person to person. Although it is possible toprovide consumers with tutorials on how to use products, it is far moredifficult to learn whether a recommended process is being followed, orwhether the recommended process is appropriate for an individualconsumer.

To learn more about consumer routines, personal care companies willsometimes conduct surveys. However, these surveys often have limitedutility. Although consumers are familiar with their own routines on apersonal level, it is often difficult for consumers to accuratelyself-report their own routines to the level of detail that is necessaryto evaluate those routines for possible improvements.

Consumer products companies will sometimes invite groups of consumers toparticipate in observation sessions so that the consumers' routines canbe studied more carefully. However, these sessions are expensive toadminister and inconvenient for the consumers as well as the companiesthat administer them. In addition, the artificial environment of theobservation sessions can lead consumers to modify their behaviorrelative to their normal routine, which can defeat the purpose of thestudy.

Although some attempts have been made to address these problems withtechnology, significant technical problems remain. For example,observation sessions may involve, with the consumers' consent, digitalvideo recordings and computerized data collection and analysis. However,while video recordings of consumer routines may be of some use to humanresearchers that review them, it is difficult to gather enoughinformation in this way to draw useful conclusions about overallconsumer preferences and behavior, because a human observer must observereach video recording to interpret what the video recording shows.

Described embodiments overcome these drawbacks and offer numeroustechnical benefits. Described embodiments can be used to provideautomated feedback on personal care routines, such as recommended stepsin a makeup application process. Described embodiments also can be usedto assist in matching a consumer with products that are appropriate forthat consumer, thereby enhancing the consumer's experience. In describedembodiments, customized information in the form of productrecommendations, recommended routines, or other information may beprovided to a user based on analysis of an uploaded video of the user'spersonal care routine. The video analysis may include machine learningor artificial intelligence techniques (e.g., template matching, edgedetection and matching, feature extraction and matching, classification,artificial neural networks, deep learning architectures, geneticalgorithms, or the like) to perform object recognition. The videoanalysis of personal care routines may be used to determine informationsuch as products used, order of usage, duration of use for each product,and how the products are being used (e.g., gestures of use). Thesedeterminations and the resulting recommendations also may be guided bymachine learning or artificial intelligence techniques.

Information provided by the user may include product information,personal information such as skin or hair attributes, satisfactionratings for products and routines, or the like. Information provided bythe user may be collected via a browser or a dedicated applicationexecuting on a computing device, and customized product information maybe received by the same computing device or some other computing device.To further improve or enhance the customized product information,additional information relating to the user could also be used, such astechnical analysis of one or more images of the user to measure or mapwrinkles, pigmentation, etc., of the user's skin. Data regarding theuser or the user's routine can be compared with other users, and theresults of those comparisons may be provided to the user or stored forresearch purposes (e.g., to improve knowledge of consumer routines orinter-product compatibility).

The customized information provided to the user may includerecommendations on changes to the user's routine, recommended products,or other feedback. The customized product information may be furtherbased on additional information such as users' answers to questionsabout themselves, their routine, or the products they use. Thecustomized product information, demographic comparisons, or otherresults may be shared by the user via a social network.

FIG. 1 is a block diagram that illustrates an example embodiment of asystem according to various aspects of the present disclosure. As shown,a system 100 includes components that allow an end user to obtaincustomized information, such as customized personal care routine datainvolving personal care products. Customized information may be providedin the form of product recommendations, tutorial videos, personal careroutine simulations, or other user experiences. Consumers can use thesystem 100 to obtain customized product information for personal careproducts that may be available in a variety of colors, textures, and thelike. Non-limiting examples of such products include eye liners, lipglosses, lipsticks, makeup, and nail polish. Although some aspects areillustrated and described as relating to personal care products, in someembodiments, products other than personal care products may be processedby the system 100. Furthermore, although some aspects are illustratedand described as relating to a personal care routine, in someembodiments activities other than personal care routines may beprocessed by the system 100.

As shown, the system 100 includes a backend product computer system 102,and a client computing device 104 (e.g., a smart phone, tablet computer,etc.). Each of these components may communicate with each other via anetwork 90, which may include any suitable communication technologyincluding but not limited to wired technologies such as DSL, Ethernet,fiber optic, USB, and Firewire; wireless technologies such as WiFi,WiMAX, 3G, 4G, LTE, and Bluetooth; and the Internet.

In some embodiments, the client computing device 104 may be used by auser to interact with other components of the system 100. In someembodiments, the client computing device 104 is a mobile computingdevice such as a smart phone or a tablet computing device. However, anyother suitable type of computing device capable of performing functionsdescribed herein, including but not limited to a desktop computingdevice, a laptop computing device, and a smart watch, may be used.

In the example shown in FIG. 1, the backend product computer system 102includes a product recommendation engine 112, an interaction analysisengine 114, a product data store 120, and an optional care routinesimulation engine 116. In some embodiments, the product recommendationengine 112 receives product information from the product data store 120along with information from the client computing device 104, and usesthis information to generate product recommendations. For example, theclient computing device 104 may provide product information obtainedfrom a product (e.g., a cosmetics product such as lipstick, lip gloss,eye shadow, foundation, or the like), along with digital video of apersonal care routine. The product recommendation engine 112 may, inresponse, use this information to develop or obtain customizedinformation (e.g., makeup application process steps, recommendedproducts, recommended shades or styles of a product, etc.) to provide tothe client computing device 104. Output from the product recommendationengine 112 may be provided to the optional care routine simulationengine 116, which can generate computer-animation simulations of theuser's care routine or a recommended care routine, or some othersimulation.

Information about a product being used can be obtained in any suitablemanner. For example, a user may be prompted during an initial setupphase to provide information about the products to be used in a personalcare routine. The user can provide product information via any suitablemethod (e.g., voice commands, gestures, text, or the like).

In some embodiments, personal care products include graphical codes,radio-frequency identification (RFID) chips, or other means for encodingpersonal care products with information that can be decoded and used togenerate customized product information. In such embodiments, the clientcomputing device 104 or some other computing device may obtain productinformation by decoding information encoded on or in the product (e.g.,in a graphical code such as a bar code or QR code, or in a near-fieldcommunication (NFC) or RFID chip). In the latter case, the clientcomputing device 104 may include or be communicatively coupled tospecialized hardware for short-range radio-frequency communication, suchas an NFC transceiver or RFID reader.

FIG. 1 depicts an illustrative scenario in which a client computingdevice 104 is used to capture video of a personal care routine. In thisillustrative scenario, a client application executing on the computingdevice 104 manages a process of recording the personal care routine,which involves application of three personal care products: eyeshadow130A, foundation 130B, and lipstick 130C. The process may involve aninitial setup in which the client application instructs the user to makebasic preparations for recording. For example, the client applicationmay instruct the user to position the camera so that all the products tobe used are in the field of view of the camera. As another example, theclient application may instruct the user to place the products in aparticular arrangement (e.g., from the user's left to right), scangraphical codes or chips on the products, or otherwise obtainidentifying information for the products, which can help the systemidentify the products correctly and follow what the user is doing duringher routine. The client application may also provide other guidance,such as instructing the user to open containers and get brushes ready,remove extraneous objects from the field of view, or make otherpreparations to avoid unnecessary movements or other visual noise duringrecording. These steps can help the system by reducing the number ofinteractions to be interpreted by the system.

The video recording may be initiated by the user's interaction with theclient application, such as by tapping a “record” button in a userinterface, or issuing a voice command. The client computing device 104can then upload the recorded video to the backend product computersystem 102 for analysis.

The backend product computer system 102 can then process the uploadedvideo in the interaction analysis engine 114 in order to analyze therecorded routine. This analysis can be performed in different ways. Forexample, one or more object recognition techniques (e.g., templatematching, edge matching, feature matching, artificial neural networks,deep learning architectures, genetic algorithms, or the like) can beused to detect which products the user is interacting with, and how theyare being used. This task may be assisted by information obtained duringthe initial setup. For example, if the client computing device 104 scansa graphical code 132A on the eyeshadow 130A and extracts a correspondingproduct identifier, this identifier can be transmitted by the clientcomputing device 104 to the backend product computer system 102 to beused in a subsequent lookup operation. In some embodiments, the backendproduct computer system 102 uses a product identifier received from theclient computing device 104 to look up information about the product(e.g., color, shape, and size of the packaging) in the product datastore 120. This lookup operation can be used to provide the interactionanalysis engine 114 with a 2D or 3D model or description of the productand/or related objects (e.g., brushes, applicators, or the like) toassist in object recognition and detection of interactions withrecognized objects. A similar process can be used for tracking otherproducts or objects that are used in the care routine. Alternatively,products or objects can be recognized in the video in other ways, suchas by assigning a particular location for specific products andinstructing the user to keep those products in the assigned locations toenable the interaction analysis engine 114 to track their use.

Once the products in the video have been recognized, the interactionanalysis engine 114 can detect physical interactions with those productsthat are performed during a personal care routine. In this way, theinteraction analysis engine 114 can identify which products were used,the order in which the products were used, the number of times theproducts were used, and/or how the products were used. In someembodiments, the interaction analysis system distinguishes care routineinteractions that will affect the end result of the care routine, suchas applying a product to the user's face, from insignificant actions,such as simply touching or repositioning a product. For example, theinteraction analysis system may use object recognition techniques todetermine whether an object, such as a brush or applicator, is in theuser's hand. The system may also use gesture recognition techniques(e.g., comparing detected hand movements to gesture models in 3Dmodel-based, skeletal-based, or appearance-based recognition algorithms)to recognize specific gestures or other actions by the user (e.g.,applying lipstick or makeup to the user's face).

Care routine interactions may be detected in different ways, dependingon the product and the level of precision of detection that may bedesired. For example, a lipstick application may be detected when thelipstick 130C is grasped and lifted to the user's face. This approachmay be easier to implement than more precise gesture detection, but itmay also introduce a risk of false positives if the user often graspsand lifts the lipstick without actually applying it to her face. Ifgreater precision is desired, more detailed recognition of a careroutine interaction may be used, such as by detecting a specificmovement of lipstick on the user's face using gesture recognitiontechniques. As another example, an eyeshadow interaction or foundationinteraction may be detected when a brush contacts a container ofeyeshadow 130A or foundation 130B and is subsequently lifted to theuser's face, indicating that the product was actually applied. In thisway, the system can distinguish from other interactions, such as where abrush is simply placed on the table next to the container. If the user'sface is not in the field of view, the system may detect an applicationwhen a brush or lipstick container is lifted out of the field of view.In this way, the system can detect care routine interactions even if theuser's face is not in the field of view, as may be necessary due toprivacy concerns or for technical reasons, such as where the camera mustbe placed close to the user's work area due to space limitations.

In the example shown in FIG. 1, the interaction analysis engine 114provides descriptions or logs of the detected interactions of a personalcare routine to the product recommendation engine 112 for furtherprocessing. In some embodiments, the product recommendation engine 112compares the detected interactions with a recommended routine andprovides customized product information, such as a suggested revision tothe routine, to the user. The product recommendation engine 112 also mayuse other information (e.g., a user ID or a product identifier extractedfrom a graphical code) to obtain further information about the productor the user. This additional information may be used to generateadditional customized product information, such as recommendations onwhether particular colors or product types should be used incombination, or whether the products used in the care routine areappropriate for a user's skin type.

Consider an illustrative scenario in which the interaction analysissystem 114 detects a personal care routine consisting of 1 lipstickapplication, followed by 2 eyeshadow applications, followed by 2foundation applications. This information can be provided to the productrecommendation engine 112, which compares the detected routine with arecommended routine of 5 foundation applications, followed by 2eyeshadow applications, followed by 1 lipstick application. Therecommended routine may be specific to the user (e.g., based on age,skin type, etc.), or a typical routine for a larger class of users.Based on the comparison, the product recommendation engine 112determines that the user applied the products in the wrong order, andthat the user did not use enough foundation.

Once this determination has been made, the product recommendation engine112 can generate customized personal care routine data for the user,which may include tutorial videos showing a recommended routine; text,audio, or graphic descriptions of suggested changes to the routine;text, audio, video, or graphical advertisements for relevant products;or the like. An optional care routine simulation engine 116 can be usedto generate customized personal care routine data in the form of, e.g.,computer-animation simulations showing a recommended routine;side-by-side comparisons with simulations that compare the user'sdetected routine with the recommended routine; or the like.

FIG. 2 is a block diagram that illustrates an example embodiment of aclient computing device 104 according to various aspects of the presentdisclosure. In the example shown in FIG. 2, the client computing device104 includes a camera 250 and a client application 260. The clientapplication 260 includes a user interface 276, which may includeinteractive functionality such as guides, tutorials, simulations, orproduct exploration technology. Product exploration technology may, insome embodiments, allow consumers to try different looks, perform colormatching, compare products with other products, test variations incharacteristics such as coverage, color, finish, etc. In an embodiment,the user interface and related technology is provided on a consumer'scomputing device (e.g., smart phone, tablet computer, etc.)Alternatively, described functionality can be provided on some othercomputing device, such as a device with a larger screen at a point ofsale for cosmetics products.

In the example shown in FIG. 2, the client computing device 104 isconfigured to record video of a personal care routine and capture imagesof graphical codes (e.g., graphical codes 132A, 132B, 132C in FIG. 1)using the camera 250. The camera provides image data and video data tothe client application 260, which performs image preprocessing 270 todetermine how the image data should be processed. For example, theclient application 260 may use image preprocessing 270 to identify theimage data as containing a graphical code. The image data is thenprovided to a code reader 272, which extracts information from thegraphical code. In the example shown in FIG. 2, the extractedinformation includes product information. Extracted product informationmay include a product ID, product description, ingredients, batchinformation, etc.

The video data can be sent along with product information (andpotentially other information, such as a user ID, device ID, or thelike) to a communication module 278 for subsequent formatting andtransmission to the backend product computer system 102. (Other featuresof the client computing device 104 are not shown in FIG. 2 for ease ofillustration. A description of illustrative computing devices isprovided below with reference to FIG. 5.)

In an embodiment, the user interface 276 provides virtual “try on”functionality that allows the consumer to virtually apply the actualcharacteristics of that product, e.g., to an image of the user's face.The user interface 276 also may provide the ability to upload an imageof the user's face for further analysis by the backend product computersystem 102 (e.g., to provide recommendations based on the user's skintype). Product information and image data transmitted to the backendcomputer system 102 can be used to perform anonymous analytics to, e.g.,monitor consumer satisfaction. Product information transmitted to thebackend product computer system 102 may include the extracted productinformation, either alone or in combination with other information suchas user preferences or selected variations for a particular product,which may be obtained via the user interface 276. Analytics (e.g., incombination with machine learning processes) can help to improve futurerecommendations for care routines as well as to provide customers withdesirable products in terms of customized tints, textures, or otherfeatures.

Within components of the system 100, or by components of the system 100working in combination, numerous technical benefits are achieved. Forexample, the ability to automatically recognize and quantify aspects ofpersonal care routines overcomes technical limitations of previousautomated consumer surveys, which provided some benefits of automationbut relied on consumers to self-report their own routines, often withinaccurate results. Additional benefits may be realized by movingprocessing burdens (e.g., video analysis, object recognition, and thelike) to the backend product computer system 102 (which may be arelatively high-powered and reliable computing system) from the clientcomputing device 104, thus improving performance and preserving batterylife for functionality provided by the client computing device 104.

FIG. 3 is a flowchart that illustrates an example embodiment of a method300 of providing customized personal care routine data for presentationin a user interface according to various aspects of the presentdisclosure. Though some embodiments of the method 300 may be used withany type of product, some embodiments of the method 300 are particularlysuitable for cosmetics products as described herein.

From a start block, the method 300 proceeds to block 302, where acomputer system (e.g., backend product computer system 102) obtainsdigital video data of at least one physical consumer product being usedin a personal care routine. In an embodiment, the backend productcomputer system 102 receives the digital video data from a clientcomputing device, such as a consumer's smartphone. The method proceedsto block 304, where the backend product computer system 102 analyzes thedigital video data. The method proceeds to block 306, where the backendproduct computer system 102 detects at least one physical interactionwith the at least one physical consumer product in the personal careroutine based at least in part on the analysis of the digital videodata. In an embodiment, the backend product computer system 102 employsone or more object recognition techniques described herein in analyzingthe video data and detecting physical interactions with the product(e.g., applications of lipstick, foundation, eyeshadow, or the like).The method proceeds to block 308, where the backend product computersystem 102 causes customized personal care routine data to be presentedin a user interface. The customized personal care routine data is basedon the detected physical interactions with the product.

In some embodiments of the methods described herein, the backend productcomputer system 102 generates the customized personal care routine databased at least in part on the at least one physical interaction andtransmits the customized personal care routine data to a client computersystem such as a smartphone, tablet computer, or the like. In response,the client computer system displays or otherwise presents the customizedpersonal care routine data in a user interface.

FIG. 4 is a flowchart that illustrates an example embodiment of a method400 of presenting customized personal care routine data in a userinterface of a client computer system (e.g., a consumer's smartphone ortablet computer) according to various aspects of the present disclosure.Though some embodiments of the method 400 may be used with any type ofproduct, some embodiments of the method 400 are particularly suitablefor cosmetics products as described herein.

From a start block, the method 400 proceeds to block 402, where theclient computer system (e.g., client computing device 104) capturesdigital video data of at least one physical consumer product being usedin a personal care routine. The method proceeds to block 404, where theclient computing device 104 transmits the digital video data to a remotecomputer system (e.g., backend product computer system 102). The methodproceeds to block 406, where the client computing device 104 receivescustomized personal care routine data from the backend product computersystem 102. The method proceeds to block 408, where the client computingdevice 104 presents the customized personal care routine data in a userinterface. At least one physical interaction with the product isdetected by the backend product computer system 102 based at least inpart on analysis of the digital video data. Alternatively, the analysisand detection can be performed by the client computing device, in whichcase the client computing device can generate or obtain the customizedpersonal care routine data on its own, and the steps of transmitting andreceiving data from the remote computer system can be omitted. Thecustomized personal care routine data is based at least in part on thedetected physical interaction(s).

In some embodiments of the methods described herein, the customizedpersonal care routine data comprises a computer-animation simulation ofa personal care routine. To generate the simulation, the backend productcomputer system 102 may represent the product as an object in a personalcare routine simulation engine and represent the detected physicalinteraction as an interaction with the object in the personal careroutine simulation engine, which can then be depicted in the simulation.

In some embodiments of the methods described herein, the backend productcomputer system 102 also obtains product information for the product,and the customized personal care routine data is further based on theproduct information. The product information may be transmitted to thebackend product computer system 102 by a client computing device alongwith the video data.

In some embodiments of the methods described herein, the productinformation is initially obtained by the client computing device 104,such as by extracting the product information from a graphical code onthe product, or obtaining the product information from the product viashort-range radio-frequency communication (e.g., NFC or RFIDcommunication).

In general, the word “engine,” as used herein, refers to logic embodiedin hardware or software instructions, which can be written in aprogramming language, such as C, C++, COBOL, JAVA™, PHP, Perl, HTML,CSS, JavaScript, VBScript, ASPX, Microsoft .NET™, and/or the like. Anengine may be compiled into executable programs or written ininterpreted programming languages. Software engines may be callable fromother engines or from themselves. Generally, the engines describedherein refer to logical modules that can be merged with other engines,or can be divided into sub-engines. The engines can be stored in anytype of computer-readable medium or computer storage device and bestored on and executed by one or more general purpose computers, thuscreating a special purpose computer configured to provide the engine orthe functionality thereof.

As understood by one of ordinary skill in the art, a “data store” asdescribed herein may be any suitable device configured to store data foraccess by a computing device. One example of a data store is a highlyreliable, high-speed relational database management system (DBMS)executing on one or more computing devices and accessible over ahigh-speed network. Another example of a data store is a key-valuestore. However, any other suitable storage technique and/or devicecapable of quickly and reliably providing the stored data in response toqueries may be used, and the computing device may be accessible locallyinstead of over a network, or may be provided as a cloud-based service.A data store may also include data stored in an organized manner on acomputer-readable storage medium, as described further below. One ofordinary skill in the art will recognize that separate data storesdescribed herein may be combined into a single data store, and/or asingle data store described herein may be separated into multiple datastores, without departing from the scope of the present disclosure.

FIG. 5 is a block diagram that illustrates aspects of an exemplarycomputing device 500 appropriate for use with embodiments of the presentdisclosure. While FIG. 5 is described with reference to a computingdevice that is implemented as a device on a network, the descriptionbelow is applicable to servers, personal computers, mobile phones, smartphones, tablet computers, embedded computing devices, and other devicesthat may be used to implement portions of embodiments of the presentdisclosure. Moreover, those of ordinary skill in the art and others willrecognize that the computing device 500 may be any one of any number ofcurrently available or yet to be developed devices.

In its most basic configuration, the computing device 500 includes atleast one processor 502 and a system memory 504 connected by acommunication bus 506. Depending on the exact configuration and type ofdevice, the system memory 504 may be volatile or nonvolatile memory,such as read only memory (“ROM”), random access memory (“RAM”), EEPROM,flash memory, or similar memory technology. Those of ordinary skill inthe art and others will recognize that system memory 504 typicallystores data and/or program modules that are immediately accessible toand/or currently being operated on by the processor 502. In this regard,the processor 502 may serve as a computational center of the computingdevice 500 by supporting the execution of instructions.

As further illustrated in FIG. 5, the computing device 500 may include anetwork interface 510 comprising one or more components forcommunicating with other devices over a network. Embodiments of thepresent disclosure may access basic services that utilize the networkinterface 510 to perform communications using common network protocols.The network interface 510 may also include a wireless network interfaceconfigured to communicate via one or more wireless communicationprotocols, such as WiFi, 2G, 3G, LTE, WiMAX, Bluetooth, and/or the like.

In the exemplary embodiment depicted in FIG. 5, the computing device 500also includes a storage medium 508. However, services may be accessedusing a computing device that does not include means for persisting datato a local storage medium. Therefore, the storage medium 508 depicted inFIG. 5 is represented with a dashed line to indicate that the storagemedium 508 is optional. In any event, the storage medium 508 may bevolatile or nonvolatile, removable or nonremovable, implemented usingany technology capable of storing information such as, but not limitedto, a hard drive, solid state drive, CD ROM, DVD, or other disk storage,magnetic cassettes, magnetic tape, magnetic disk storage, and/or thelike.

As used herein, the term “computer-readable medium” includes volatileand non-volatile and removable and non-removable media implemented inany method or technology capable of storing information, such ascomputer readable instructions, data structures, program modules, orother data. In this regard, the system memory 504 and storage medium 508depicted in FIG. 5 are merely examples of computer-readable media. In anembodiment, computer-readable media can be used to store data for use byprograms.

Suitable implementations of computing devices that include a processor502, system memory 504, communication bus 506, storage medium 508, andnetwork interface 510 are known and commercially available. For ease ofillustration and because it is not important for an understanding of theclaimed subject matter, FIG. 5 does not show some of the typicalcomponents of many computing devices. In this regard, the computingdevice 500 may include input devices, such as a keyboard, keypad, mouse,microphone, touch input device, touch screen, tablet, and/or the like.Such input devices may be coupled to the computing device 500 by wiredor wireless connections including RF, infrared, serial, parallel,Bluetooth, USB, or other suitable connections protocols using wirelessor physical connections. Similarly, the computing device 500 may alsoinclude output devices such as a display, speakers, printer, etc. Sincethese devices are well known in the art, they are not illustrated ordescribed further herein.

While illustrative embodiments have been illustrated and described, itwill be appreciated that various changes can be made therein withoutdeparting from the spirit and scope of the invention.

The invention claimed is:
 1. A computer-implemented method, comprising,by a computer system comprising one or more computing devices: obtainingdigital video data of at least one physical consumer product being usedin a first personal care routine; analyzing the digital video data usingobject recognition and gesture recognition; detecting at least onephysical act of using the at least one physical consumer product in thefirst personal care routine based at least in part on the analysis ofthe digital video data; comparing the detected at least one physical actwith at least one physical act in a second personal care routine;determining a recommended change to performance of the detected at leastone physical act in the first personal care routine based at least inpart on the comparing; generating customized personal care routine datacomprising the recommended change to the performance of the detected atleast one physical act in the first personal care routine; and causingthe customized personal care routine data to be presented in a userinterface.
 2. The method of claim 1, wherein the at least one physicalconsumer product comprises two or more physical consumer products. 3.The method of claim 1 further comprising obtaining product informationfor the at least one physical consumer product, wherein the customizedpersonal care routine data is further based on the product information.4. The method of claim 3, wherein the product information is extractedfrom a graphical code on the at least one physical consumer product. 5.The method of claim 1, wherein causing the customized personal careroutine data to be presented in the user interface comprises:transmitting the customized personal care routine data to a clientcomputer system, wherein the user interface is implemented in the clientcomputer system.
 6. The method of claim 1 further comprising:representing the at least one physical consumer product as an object ina personal care routine simulation engine; and representing the detectedat least one physical act of using the at least one physical consumerproduct as an interaction with the object in the personal care routinesimulation engine.
 7. The method of claim 6, wherein the customizedpersonal care routine data further comprises a computer-animationsimulation of a personal care routine generated by the personal careroutine simulation engine.
 8. The method of claim 1, wherein the atleast one physical consumer product comprises a cosmetics product.
 9. Acomputer system comprising: a camera; a processor; and computer-readablemedia having stored thereon instructions configured to cause thecomputer system to perform steps comprising: capturing digital videodata of at least one physical consumer product being used in a firstpersonal care routine; analyzing the digital video data using objectrecognition and gesture recognition; detecting at least one physical actof using the at least one physical consumer product in the firstpersonal care routine based at least in part on the analysis of thedigital video data; comparing the detected at least one physical actwith at least one physical act in a second personal care routine;determining a recommended change to performance of the detected at leastone physical act in the first personal care routine based at least inpart on the comparing; obtaining customized personal care routine datacomprising the recommended change to the performance of the detected atleast one physical act in the first personal care routine; andpresenting the customized personal care routine data in a user interfaceof the computer system.
 10. The computer system of claim 9, wherein theat least one physical consumer product comprises two or more physicalconsumer products.
 11. The computer system of claim 9, wherein theinstructions are further configured to cause the computer system tocapture a digital image of a graphical code on the at least one physicalconsumer product or obtain product information from the at least onephysical consumer product via short-range radio-frequency communication.12. The computer system of claim 9, wherein the customized personal careroutine data further comprises a computer-animation simulation of apersonal care routine.
 13. The computer system of claim 9, wherein thecustomized personal care routine data further comprises a comparisonwith other users.
 14. The computer system of claim 9, wherein the atleast one physical consumer product comprises a cosmetics product. 15.The computer system of claim 9, wherein the computer system comprises asmartphone.
 16. A non-transitory computer-readable medium having storedthereon instructions configured to, when executed by one or morecomputing devices of a computer system, cause the computer system toperform steps comprising: obtaining digital video data of a plurality ofphysical consumer products being used in a first personal care routine;analyzing the digital video data using object recognition and gesturerecognition; detecting physical acts of using the plurality of physicalconsumer products in the first personal care routine based at least inpart on the analysis of the digital video data; comparing the detectedphysical acts with a sequence of interactions in a second personal careroutine; determining a recommended change to performance of the detectedphysical acts in the first personal care routine based at least in parton the comparing; generating customized personal care routine datacomprising the recommended change to the performance of the detectedphysical acts in the first personal care routine; and transmitting thecustomized personal care routine data to a client computer system,thereby causing the customized personal care routine data to bepresented in a user interface of the client computer system.
 17. Thecomputer-readable medium of claim 16, the steps further comprisingobtaining product information for at least one of the plurality ofphysical consumer products, wherein the customized personal care routinedata is further based on the product information.
 18. Thecomputer-readable medium of claim 16, wherein the recommended change toperformance of the detected physical acts comprises a recommended changeto the order of the detected physical acts.
 19. The computer-readablemedium of claim 16, wherein the recommended change to performance of thedetected physical acts comprises a recommended change to a product beingused in at least one of the detected physical acts.